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^ ' Abstract. An infinite-dimensional version of Calogero-Moser opera- 

tor of BC-type and the corresponding Jacobi symmetric functions are 
introduced and studied, including the analogues of Fieri formula and 
Okounkov's binomial formula. We use this to describe all the ideals lin- 
early generated by the Jacobi symmetric functions and show that the 
deformed BC{m, n) Calogero-Moser operators, introduced in our earlier 
work, appear here in a natural way as the restrictions of the BCoa oper- 
ator to the corresponding finite-dimensional subvarieties. As a corollary 
rG ' we have the integrability of these quantum systems and all the main 

jrt I formulas for the related super Jacobi polynomials. 
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1. Introduction 

The classical Jacobi polynomials P" (t) (sometimes called also as hy- 
pergeometric polynomials) are the orthogonal polynomials with respect to 
the weight (1 — t)°(l -|- t)^ on the interval [—1, 1]. They satisfy the Jacobi 
differential equation 

(1 - t^)y" + [P-a-{a + l3 + 2)t]y' + n{n + a + P + l)y = 0, (1) 

and thus are particular case of the hypergeometric functions. 
Replacing the weight by 



n \ii-h\^' n i^-uni+u 



fi 



l<i<j<n l<i<n 
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on the n-dimensional cube we come to the multidimensional Jacobi polyno- 
mials J'x{t;6,a,f3) (see e.g. [12], where the parameters a,/3 are denoted by 
a, b). After change of coordinates and gauge transformation these polynomi- 
als are the eigenfunctions of the BCn version of Calogero-Moser-Sutherland 
(CMS) operator [13] and for special values of parameters can be interpreted 
as spherical functions on some compact symmetric spaces, including Grass- 
mannians. 

The BCn CMS operator is a particular case of the larger class of the so- 
called deformed CMS operators introduced in |16j in relation to generalised 
root system of type BC{m, n) : 



m 



L- = -A„-tA„ + V(^¥^±iL+ 2t(* + l 



.m n.^n^ l^y 2 



. ^sinh {xi — Xj) sinh {xi + Xj) 



n 



, y ( 2(fc-i + 1) ^ 2(fc-i + 1) 
^ V sinh^ {yi - yj ) sinh^ (y^ + y^ ] 



yyf 2(fc + l) ^ 2(fc + l) \ ^^ p{p + 2q+l) 
~[~[\ sin^ {xi - yj ) sinh^ {xi + yj) J {r{ sinh^ x, 

y. 4g(g+l) ^ " A;r(r + 2g + l) ^ ^ Aks{s + l) ^^^ 

~^ sinh 2xi —^ sinh yj ~^ sinh 2yj 



where 



and the parameters k,p,q,r,s satisfy the following relations 

p = kr, 2q + l = k{2s + l). (3) 

The usual BCm CMS operator corresponds to the case n = 0, when there is 
no variables y. The relation with the standard Jacobi parameters is 
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a = -p-q--^f3=-q-- (4) 

and 9 = —k/2. In the special case n = l, p = r = these operators were 
introduced in [Sj SJ [7], where the integrability of the corresponding quan- 
tum problems was established within the theory of Baker- Akhiezer function. 
In case when both m and n are larger than 1 this method can not be ap- 
plied and the integrability was proved in [1§l\ by a direct (but complicated) 
construction of the integrals. In the case An^m we gave in |17j a more concep- 
tual proof of the integrability by showing that the corresponding deformed 
CMS operator can be described as the restriction of the usual CMS operator 
with infinite number of variables onto certain subvarieties called generalised 
discriminants. 

The main goal of this paper is to extend this approach to the BC-ca.se. 
This is not straightforward since in contrast to the A-case there is no stability 
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for the CMS operators of BCn type. Our SCoo-version of Calogero-Moser 
operator depends on an additional parameter h, which is related to the 
dimension A^. In the power sums coordinates Pa it has the form 



■a-2 

'^Pa-b-li'^Pb+Pb^ 
.b=0 



da 



a,b>0 a=2 

oo 

+ ^ [(a + k{a + 1) + 2h)pa + (2a - 1 + 2ka + 2h- p)pa-i] da, 

a=l 

where 

Po = -k'^{h+-p + q) 

is the substitute for the dimension and we use the notation da = Qg^ (cf- 
[U [IB]). This operator can be naturahy represented as the sum of the usual 
Aoa CMS operator with an additional /i-dependent momentum term and the 
rational BC^ Calogero-Moser operator (see Section 3). 

We show that this 4-parametric family /^('^'P'?'") has symmetry group Z2 
generated by 3 commuting involutions (see Theorem 13.21 below) ) . One of 
them is the extension of the duality k -^ k~^ well-known in A^^ case jTO] . 
It acts on other parameters as 

2h-l = k-^{2h -l),p = k-^p, {2q + 1) = k-^{2q + 1), 

and thus gives an explanation of the relations ([3|) in the definition of the 
deformed CMS operator. Note that we can not have this symmetry at the 
finite-dimensional level. 

These involutions lead to the dualities for the Jacobi symmetric functions 
J'\{u;k,p,q,h), which are defined as certain eigenfunctions of the operator 
£{k,p,q,h)^ These functions are the limiting case of the Koornwinder polyno- 
mials in infinitely many variables introduced by Rains [15j . 

We present the infinite dimensional analogues of Fieri formula, which in 
BCn case was found by van Diejen [5], and Okounkov's binomial formula, 
expressing Jacobi polynomials in terms of Jack polynomials [llj . We use 
these formulas to describe all ideals, which are linearly spanned by the Jacobi 
symmetric functions {BC -invariant ideals) and show that the deformed CMS 
operator ([2]) after some gauge transformation is the restriction of the BC^o 
operator onto the corresponding subvariety, which implies integrability of 
the corresponding quantum system and a deformed version of the Harish- 
Chandra homomorphism. 

The image of the Jacobi symmetric functions under this restriction gives 
the eigenfunctions of the corresponding commuting differential operators, 
which are called the super Jacobi polynomials. We show that these polyno- 
mials have properties similar to Jacobi polynomials by presenting deformed 
versions of Fieri and Okounkov's formulas as well Opdam's evaluation for- 
mula [14|. Remarkably all these formulas can be written naturally in terms 
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of the corresponding deformed root systems, which is another evidence of 
the importance of this notion. 

The Fieri formula can be interpreted as bispectraUty between BC{m, n) 
CMS operator ^ and a deformed version of the rational Koornwinder op- 
erator (see formulas (|92]) . (f94]) below). In the special case n = and n = 1 
this bispectrality was established earlier by Chalykh [2] and Feigin [8], who 
used the technique of Baker- Akhiezer functions. 

Although our initial motivation came from the theory of quantum in- 
tegrable systems we believe that the most interesting applications of our 
results are in representation theory of orthosymplectic Lie superalgebras 
and related symmetric superspaces. We were also partly stimulated by an 
important paper by Okounkov and Olshanski |12j . where the asymptotic 
behaviour of the Jacobi polynomials J\ as the number of variables and the 
Young diagram A go to infinity, was investigated. 

2. Jacobi polynomials for the root system BCn 

Let P/v = C[x^ , . . . ,Xjy ] be the algebra of Laurent polynomials in N 
independent variables and W be the BCn Weyl group W = Sn x "^2 ■ We 
have a natural action of the group W on Pjy: Sn permutes the variables 
rri, . . . , xn while Z^ acts by j;j — s- x" , z = 1, . . . , A^. Let P^ C Pn be 
the subalgebra of the corresponding Ty-invariant Laurent polynomials; it is 
well-known that it can be identified with the standard algebra of symmetric 
polynomials (see below). 

Recall that the generalised CMS operator related to any root system R 
has the following form [13] 



ma{ma + 2m2a + l)(a, a) 
"7^ sinh^(a,2:) 



T A , Y^ ma{ma + ^m2a + i-m,a) ^ 

j-'R — —^+ / , . . o, — ^ , z e M , 



which is gauge equivalent to 

£ij = — A -|- 2 N^ ?7Za coth(Q, z)9q,) 

aG-R+ 

where da = (c^i ^) s-i^d ma are some parameters ("multiplicities") pre- 
scribed to each root a in VF-invariant way, W is the corresponding Weyl 
group. In the case of BCn I'oot system 

R = {±ei, ±2e,i = l,...,A, ie^iej, 1 < i < j < A}, 

where Ei is the standard basis in M , we have the following operator 

i=\ l<i<i<A ^^* ^J' ^*^J' ^ ^ 

7 + 2^4 — - \xidi, 

Xi-1 xf-lj 



E U 



(5) 



where Xi = exp 2zi, i = 1, . . . ,N and k,p,q are the corresponding multipHc- 
ities 

m(ibej) = p, m{±2ei) = q, m{±£i ± Ej) = k. 

Note that our multiphcities are different from the usual ones by a sign 
change. 

The Jacobi polynomials V\{x, k,p, q) (in the Laurent form) are the eigen- 
functions of this operator labelled by partitions A. Recall that a partition is 
any sequence 

A = (Ai, A2, . . . , Ar . . . ) 

of nonnegative integers in decreasing order 

Ai > A2 > • • • > Ar > . . . , 

which contains only finitely many nonzero terms. The number of nonzero 
terms is called the length of A and denoted by /(A). The sum 



A 1= A1 + A2 + ... 



is called the weight of A. 



In algebra PW there is the following natural analogue of the monomial 



N 
symmetric functions: 



mx{xi,...,XN) = y^x"': 



.0.2 ^N 



where A is a partition with /(A) < N and the sum is taken over all distinct 
sequences ai, . . . , a^ such that |ai|, . . . , \an\ is a permutation of A. It is easy 
to see that such m\ form a basis of P^ . 

Consider the following partial order on the partitions. We write /u < A if 
and only if for all i = 1, 2, . . . we have 

Ai H \- Xi < ni-\ h^i- 

The Jacobi polynomials Vx{x,k,p,q), which usually are defined by the 
orthogonality condition (see e.g. [12j), can also be defined in the following 

way [6, 9j. 

Theorem 2.1. Ifl,k,p + 2q are linearly independent over rational numbers 
then for any partition A , /(A) < A^ there exists a unique Laurent polynomial 
V\{x,k,p,q) G P^ such that 

1) Vxix,k,p,q) = mx + J2f,<x'^Xt,'m^ for some ux^, = uxi,{k,p,q) 

2) Vx{x,k,p,q) is an eigenfunction of the operator C^^'. 

Proof. Let Vx is a subspace in P^ formed by linear combinations of m^ 
where /i < A. It is known [9j that the operator C^ ' has an upper triangular 
matrix in the basis m,. 



"ix ■ 



jr(^\mx) = Y.cx,r 

fi<X 
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where the coefficients cx^ can be described exphcitly [6j . In particular, 

N N 

cx,x = E ^? - 2^ E(^ - ^)^^ - (P + 29)1^1 = 

2n(A') + 2kn{X) - \X\{p + 2q - 1 + 2k{N - 1)), 

where n(A) = ^j>i(« — l)Aj and A' is the conjugate partition (see e.g. [10]). 
We have 

cx,x-Cf,,f, = 2k[n{X)-nif,)]-2[n{fi')-niX')] + {\f,\-\X\){p+2q-l+2k{N-l)), 

which for generic k,p, q is equal to only if |A| = \n\ and n(A) = n(^). But 
it is easy to check that if |A| = |^| and /i < A, then n(A) = n{fi) only if 
A = /i. This means that ca,a t^ c^,/^ if A > /i and the theorem follows now 
from linear algebra. D 

In the new variables 

Ui = -{xi + x;^-2),i = l,...,N (6) 

the algebra 

is the standard algebra of symmetric polynomials of A^ variables and the 
Jacobi polynomials can be rewritten as symmetric polynomials 

'Px{xi,...,XN;k,p,q) = Jx{ui,...,UN;k,p,q). 

Slightly abusing terminology we call J^x{ui, ■ ■ ■ , un', k,p, q) Jacobi symmetric 
polynom,ials. Note that because of the coefficient 1/2 in formula ^ the 
coefficient at mx in the expansion of J^x is 2' ' (but not 1). Note also that 
the value Xj = 1 corresponds in the new variables to Uj = 0. 

One can check that in the new coordinates the operator ([5]) has the fol- 
lowing form 



J ^3 



^-^ ^-^ Ui — Uj ^^ Ui — Uj 

i=l i<j ■' i<j ■' 



N N 



-{p+2q-l + k{N- 1)) J2 ^idi - (2p +2q-l)^di 

i=l i=l 

(7) 
When N = 1 this leads to the Jacobi differential equation ([1]) with 

t = Ul + l. 

It is interesting to note that the operator d?]) can be represented as the 
sum 

/:w = ^w+ew, (8) 
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where 

N N N 

A(^)=J2(ui^if-k^^^^^^{uA-UJ^J)-{p+2q + k{N-l))Y,u^^^ 

. -I . . . Ui Uj 

1=1 1<J ■' 1=1 

(9) 
is the usual ^at-i CMS operator (see e.g. formula (2) in [T7]) with an 
additional momentum term, and 

g(^) = 2 ^ u^dl - 4fc ^ ^-^- " ^^-^^^ - (2v + 2g - 1) ^ a., (10) 

i=l i<j ■' 1=1 

which is the (gauged) rational BCj^i Calogero-Moser operator 

1<J ■' ■' 1 = 1 

rewritten in coordinates Ui = xf. 

In the algebra Ajv we have the usual basis consisting of monomial sym- 
metric functions 



mx{ui,...,UN) = y^ul^'' 



0,2 0]s[ 



where the sum is taken over all distinct sequences oi , . . . , a„ such that 
oi, . . . , a^i is a permutation of A. It is easy to see that the transition matrix 
from the basis {mx(ui, . . . , un)} to the basis {m\{xi, . . . , xpf)} is upper tri- 
angular, so the operator C^^' has an upper triangular matrix in the basis 
{mx{ui,... ,un)}. 

The following result due to Opdam [H] gives the value of the Jacobi 
polynomials V\{1) = V\{1, • • • , 1; k,p,q) (which in the new coordinates is 
the same as Jx{0) = Jx{0, • • • ,0, k,p,q)). 

Let 

R+ = {ei, 2ei, ei±ej,l<i<j < N} 
be the positive part of the BCn root system. Introduce 

1 ^ / 1 \ 

P = 2 X] "^"" = Yl [ ^(^ -i) + 2^ + 'l]^i- (11) 

a£R+ i=l ^ ' 

and 

^,. ^ _ r((A-p,a^) -mc,-im„/2)r((-p,a'^) - im^/2) 
r((A - p, Q^) - 2'ma/2W{{-p, a^) -ma- ifnai2) 

where q^ = ; "^ For an explicit form of ()12p in the BCn-case see Proposi- 
tion 12.31 below. 



Theorem 2.2. (Opdam [Hjj 

^^^(1)= n ^(^'")- (13) 



In order to define infinite dimensional analogs of Jacobi symmetric func- 
tions we need the following version of Opdam's formula ()13p essentially due 
to Rains [TSj. 

Introduce for any partition A the following functions of one variable 

<^^(^)= n (\+i + A:(A;+^)+x), (14) 

Cxi^)= n {^^-j-H>^'j-^)+x), (15) 

C'i{x)= n (j-l + A;(i-l)+x). (16) 

fe-)eA 

Proposition 2.3. 

Proof. Let us rewrite the right hand side of the Opdam formula l\13\i as the 
product 

Ja(0) = ^+^-^°, 
where 

^+^-1-1- r(A, + Aj + k{i+j-l-2N)-p- 2q) yr T{k{i+j-2N)-p-2q) 
-I--1: r{Xi + Xj+k{i+j-2N)-p-2q) ^^.r{k{i + j - I - 2N) - p - 2c^ 

is the part corresponding to the roots Si + Sj, ^ < J? 

T-r r(A,-A,+fc(^-j-l)) ^ r(/c(i-j)) 

J- J- r(Ai-A,+A:(i-j)) J-J- T{k{i-j-l)) 
l<i<j<N ^ * J ' V "'/^ l<i<j<N V V J ;; 

is the part corresponding to the roots £i — Ej, i < j and 

_ -TT r(2A, + 2k{i -N)-2p- 2q) -pj- T{2k{i -N)-p-2q) 



■^ r(2Ai + 2k{i -N)-p-2q) fj;^ T{2k{i -N)-2p- 2q) 
j^ T{\, + k{i-N)-p-2q) ^ T{k{i-N)-p-q) 

ii V(\- -U h(i - N\ -r,- n\ ii 



■L r(Ai + A;(i - A^) - p - g) ^^-^ T{k{i -N)-p-2q) 

is the part corresponding to the roots Si and 2ej. 
One can check that C^{x) can be rewritten as 



Aj-Aj+i 
l<i<j<Ar 



'^\i^) = n (^i + ^i+1 + ^(i + i) + a; + 1 
l<i<j<Ar 

n i(2Aj + 2fci + 3; + l) -|-r _ 

. T{\i + k{i + N) + x + l) , M .T 



T{2Xi + 2ki + x + l) -j-r r(Ai + Aj + fc(z + j)+x + l) 

L r(Ai + fc(z + AT) + a; + 1) ^<.y<^ r(A, + A,- + k{i + j - I) + x + 1) 



where we have used the standard notation {x)n = ^ix) • ^® ^ partial case 
for A = we get an identity 

^^ -j-r T{k{i + j)+x + l) -pr T{2ki + x + l) 

~ ,J^j^^nHi + J-i)+x + l)l\Tikii + N)+x + iy 

Using this identity we can rewrite the expression for A~^ as follows 

A+^ \r+( ^^-l-r^ r(2A, + 2H + x + l) T{k{i + N) + x + l) 

f}^r{Xi + k{i + N) + X + 1) r{2ki + x + i) ' 

where x = —2kN — p — 2q — 1. 
In the same way we have 

i-l + fc(z-l-iV) ^ CO(-fcJV) 
^o^TTr(2A, + 2A:i + x + l-p) r(2A;i + x + 1) 



and 



r(2Aj + 2A;i + X + 1) T{2ki + x + l-p) 

yr r{\ + k{i + N)+X + l) yj r{k{i + N)+X + q+l) 
^^T(\i + k(i + N)+x + q+l)^ T( 



f=i ^(^^ + k{i + N)+x + q+l) f\ T{k{i + N)+x + l) 

with the same x = —2kN — p — 2q — 1. 
Therefore 

V aw; ij. Y{2ki + x + l-p) 

y. T{kii + N)+x + q+l) ^^|;,| C°(fc(l-iV)-p-g+l/2) 
l}^TiXi + k{i + N)+x + q+l) C+{-2kN -p-2q-l) 

This implies the formula (jl7|) . D 

Another important result is the following Fieri formula for Jacobi poly- 
nomials, which was found by van Diejen [5]. 

Let J C / c N be two finite subsets of natural numbers and a{J) = 
{aj)j^j be a sequence of signs aj G {il}i j G J- Define the following 
functions of variables xi , where / € /: 

^J{J),i = n ^(^i^i) H v{criXi + ajXj)v{aiXi + ajXj + 1) 

JJ v{ajXj + xi)v{ajXj - xi) 
j&J, i&i\J 

^a{j),i = n wiajXj) JJ v{aiXi + ajXj)v{-aiXi - ajXj - 1) 
jeJ i<j, ijeJ 
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Y\ v{ajXj + xi)v{ajXj - xi), 
jeJ, iei\J 
where 

.^ {x-lp-q){x-^p+^) x-k 

w{x) = ^ , v{x) = — — 

x[x + ^j X 

Let us assume from now on that the parameters k,p,q are generic in the 
sense that together with 1 they are hnearly independent over Q. 

Theorem 2.4. (van Diejen [5]) For generic values of parameters Jacobi 
symmetric polynomials satisfy the following Fieri identity 

where Ej. is the r—th elementary symmetric function in variables ui, . . . , un, 
the sum is taken over all cr{L), (j{J), where L,J<ZI= {1, 2, . . . , N}, LCiJ = 
0, \L\ + \J\ = r such that X + a{J) is a partition, J^ is the complement of J 
in I and pi = kN + -^p + q — ki, i = 1, . . . ,N. 



For r = 1 the Fieri formula can be rewritten as 

N 



«".)i|^En^.«(t^-tM) 



(19) 



where 



T(A + /3,a) 



O = {isjl j = 1, . . . , N} and by A it e^ we mean the partition (Ai, . . . , Aj =b 
1, . . . , Xn) when it is indeed a partition (otherwise the corresponding term 
is to be omitted in the right hand side). GeometricaUy this corresponds to 
the adding or deleting a box from the Young diagram. In this form it can 
be generalised to the deformed case (see section 7 below). 

In the algebra of symmetric functions there is another basis consisting of 
Jack polynomials Fx(u, k) (see [ID])- A remarkable binomial formula due to 
Okounkov [111I12J expresses the Jacobi polynomials in terms of Fx{u, k). In 
order to write down this formula we need the following interpolation BCm 
polynomials I\{zi, . . . , ZN',k,h) introduced and studied by Okounkov and 
Olshanski \TT[\T2\. 



Let A]y' C C[zi, . . . , Z]\[] be the algebra of polynomials in A^ variables 
zi, . . . , zn, which are symmetric in variables {zi + ki + /i)^, i = I, . . . ,N, 
where h is an additional parameter. If 

h = -kN- -p-q (21) 

then ki + h = —pi, where p is given by (fTTI) . and the algebra Ay^' can 
be interpreted as the image under the Harish-Chandra homomorphism of 
the algebra 'D{N,k,p,q) of quantum integrals of the BCn Calogero-Moser 
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system (see |131[9]). Namely, for any polynomial / G Aj^' with h given by 
([2T]) there exists a differential operator Cj commuting with ([7]) such that 

Cf{Vx{x;k,p,q)) = fiX)Vxix;k,p,q). (22) 

Proposition 2.5. (Okounkov, Olshanski [12j j For any h and any partition 
X with /(A) < A^ there exists a unique polynomial Ix{zi, . . . , z^; k, h) E A]y' ' 
of degree 2|A|, which satisfies the following conditions: 

lx{fi;k,h)=0 

for any partition fi = (^i, . . . ,/iAr) such that A ^ /i, and normalisation 
condition 

Ix{X,k,h) = C-{l)C^{2h-l). 

Here we are using the usual geometric representation of the partitions as 
Young diagrams, so A C /x means that the inclusion of the corresponding 
sets. For example, 

TV 

I,^i-){zi,...,ZN;k,h) = '^Zi{zi + 2ki + 2h), (23) 

i=l 

where (1) denotes the Young diagram consisting of only one box. 

There exists the following combinatorial formula for these polynomials 
found by Okounkov [TT]: 

Ix{zi,...,ZN;k,h) =^ipT{-k) JJ [{zrp(^ij) + h + kT{i,j)f 

T {i,j)ex (24) 

-{(j-l) + k{i-l) + h + kT{i,j)f]. 

Here the sum is taken over all reverse tableaux T of shape A with entries 
in {1, . . . ,N}, and ipTi—k) is the same weight as for ordinary Jack poly- 
nomials (see [i2l)- From this formula we see that the leading part of the 
polynomial Ix{z; k, h) coincides with Px{zf, . . . , zj^; —k), where Px{x, —k) is 
the corresponding Jack polynomial. 

Now we are ready to present the binomial formula for Jacobi polynomials. 

Theorem 2.6. (Okounkov |1H I12j ) The Jacobi polynomials can be written 
in terms of Jack polynomials as follows: 



Jx{ui,...,UN;k,p,q) = V2'^' , , /^, ' ' Pf,{ui,...,UN;-k), (25) 

^^ Jf,{0)I^,{H■,k,h) 

where h = —kN — 2? — Q dnd 1{X) < N. 

As a corollary we have the following version of Fieri formula for interpo- 
lation BCn polynomials. Let a^,^ be the matrix coefficients of the integrals 
Lf in the basis of Jack polynomials: 

Cf{Px{u, k)) = ^ ax,^,Pf,{u, k). 

AtCA 
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They can be found from a triangular system of linear equations 
with 

(k h) 

Proposition 2.7. For any f G A^^' we have 

/(A)/^(A; k,h) = Y, C^,,a,J,{\; k, h), (26) 

where \iy\ — \n\ < 2deg{f) and the coefficients C^^u, a\,fi are defined above. 

3. Jacobi symmetric functions of infinitely many variables 

In this section we are going to generalise all the results from previous sec- 
tion to the algebra of symmetric functions in infinitely many variables. Let A 
be the projective limit of the graded algebras A^v of symmetric polynomials 
in N variables: 

A = limAAT. 

The power sums 

pi=u[+u'2 + ..., 1 = 1,2,... 
is a convenient set of free generators of this algebra, the set 

PX = PXiPX'z ■ ■ ■ 

for all partitions A forms a linear basis in A. Consider a natural homomor- 
phism ipj\f : A — > A^ defined by 

N 

^n{pi) = Y.u\. (27) 

An infinite dimensional version of the BCn CMS operator C^ ' is the oper- 
ator Ci^'P'i'^)^ depending on extra parameter h, replacing the dimension A^. 
To write it down it is convenient to introduce the notation 

da = aiT—, a = 1,2, 

OPa 

Theorem 3.1. i) There exists a unique second order differential operator 
£{k,p,q,h) ; ^ _> ^ with coefficients, which are polynomial in all the parame- 
ters, such that for all N = 1,2, .. . and h = —kN — ^p — q we have 

^No6^^'P^''^^^ = 6"'^o^^, (28) 

where C^^' is the BCn CMS operator given by ^. 
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ii) Operator C(^'P'1^^> has the following explicit form in the power sums 
coordinates pa : 

ra-2 



^ik,p,g,h) ^ Y^ (^^^^ + 2p,+,_,)dadt -kY, 



a,b>0 



a=2 



YPa-b~l{'2Pb + Pb+l) 



.6=0 



da 



+ ^ [(a + k{a + 1) + 2h)pa + (2a - 1 + 2ka + 2h- p)pa-i] da, (29) 



a=l 



with 



Po 



-k ^{h+-p + . 



(30) 



Proof. Represent the operator ()29[1 as the sum 

£(k,p,q,h) _ j^(k,h) _|_ Q{k,p,q,h) 

where 

oo /a— 2 \ oo 

_4(fcA) = ^ Pa+bdadb-kY, 2^Pa-6-iP6+i 5„+J] [(a + k^a + 1) + 2/i)p,] a„, 

a,fe>0 a=2 \b=0 / a=l 

(31) 

oo /a— 2 \ 

^(fc,P,g,h) ^ 2 ^ p,+b_i5,5, - 2A: j; j; Pa-f,-iPfe da (32) 

a,b>0 a=2 \fe=0 / 



+ ^(2a - 1 + 2A:a + 2/i - p)pa-ida, 



a=l 

where Po = —k^^{h+^p+q). One can recognise in the first operator the usual 
^oo CMS operator (see [HUB]) with an additional /i-dependent momentum 
term, while the second operator is actually the rational BCoo Calogero- 
Moser operator (cf. formula ([8])). 

To prove the theorem it is enough to show that if /i = —kN — 2P~ Q then 
for any P £ A 

and 

0(^)((^7V(P))=C^7V(S(''^'''''^)(P)), 

where the operators A^^' and B^^' are given by Q and pUj) respectively. 
Since the order of the corresponding differential operators is either one or 
two it is enough to prove this for power sums P = Pa, a = 1, 2, . . . and their 

products PaPb- 

One can check the following relations: 

a-l 

A^^Hpa) = (1 + k)a^pa - ka^PrPa-r - a{p + 2q + k{N - l))pa, 

r=Q 

A^'^HpaPb) = A^''\pa)Pb+PaA^''\pb) + 2abpa+b, 

13 



a-1 
B^^\pa) = 2(1 + k)a^pa-i - 2ka^PrPa-r-l " a(l + 2p + 2q)pa-i, 

r=0 

B^^'HpaPb) = B^''\pa)pb+PaB^''\pb) + 4abp,+k-l 

with pq = N. The proof is straightforward with the use of the following 
identities: 

y^iui + Uj)— ^ = -apa + y^ PiPa-i-, 

t<j ■' 1=0 

^ ^a _ a 1 . a-1 

= -o«Pa-l + 7T Z^PiPa^i'l, 

Uj — Ui 2 2 ^-^ 

i<j ■' i=0 



where again pQ = N. The comparison with (j3ip . (j32p proves the existence. 
The uniqueness part is evident. D 

Consider now the following automorphisms uj and 9 of the algebra A : 

uj{pi) = k'^pi, (33) 

e{p,)=p, + {-2f±^^. (34) 

Theorem 3.2. The BC ^-operator C^^'P^t'^i has the following symmetries: 

£{k,p,q,h) _ £{k,p',q',h) r^r^\ 

W o C^^'P''}^^) = kC^k-\r,s,h) ^ ^^ (3g) 

Q ^ ^{k,p,q,h) ^ ^{k,p,q,h) ^ Q^ ^37^ 

where 

p' = l + 2k-p-2q, q' = q, (38) 

2h-l = k~^{2h - 1), r = k'^p, (2s + 1) = A;"^(2g + 1), (39) 

p= -p^ q = 2k + l-q. (40) 

Proof. The symmetries (|35|) .(|36 p hold for ^C^''*) and }3(^^p^i^^> separately and 
are obvious from their form. The third symmetry holds only for the full 
operator (j29p . Let us look more generally for the shift symmetry of the 
form 

Pa ^Pa+la- 

From the second order part we see immediately that constants ^a must 
satisfy the relation 7a+i + 27^ = 0, which implies that 7^ = (—2)^7 for some 
constant 7. A simple analysis shows that this gives the symmetry of the 
operator ([29]) if and only if 

_ 2k + l-2q 
^ ~ 2k 

and the corresponding parameters are related by the formula (j40p or by the 
formula 

p = p + 2q - 2k - l,q = 2k + I - q, 
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which corresponds to the composition of 6 with the first symmetry (j38|) . D 

Remark. The symmetry ()36p is an analogue of duahty k — > k~^ in 
the theory of Jack polynomials (see [lO]). The fact that the corresponding 
change of parameters (j39p coincides with the relations ^ in the definition of 
the deformed BC{m,n) operators (similarly to the ^(?tt,, n)-case [17]) seems 
to be important and shed more light on the nature of the deformed root 
systems |16] . We note also that one can not see any of these symmetries at 
the finite-dimensional level. 

Let us define now the Jacobi symmetric functions of infinitely many vari- 
ables J\{u; k,p, q, h) G A. 

Theorem 3.3. //I, k, h are linearly independent over rational numbers then 
for any partition A there exists a unique polynomial J\{u] k,p, q,h) G A such 
that 

1) Jx{u;k,p,q,h) = 2^^^mx{u)+Y,|J,<xUx^_L'mf^{u) for someuxfj, = ux^,ik,p,q,h) 
^) J^x{u;k,p,q,h) is an eigenfunction of C^^''^''^'^' with the eigenvalue 

2n{\') + 2kn{\) + \\\{2h + 2k + 1), 

where as before n(X) = X]j>x(^ — l)Ai and X' is the conjugate partition. 



The proof is the same as in the finite-dimensional case (see Theorem 12. ip 
and uses the upper-triangularity of the operator in the monomial basis. 

Remark. As it follows from Okounkov's binomial formula (see ()64p be- 
low) the Jacobi symmetric functions Jx{u;k,p,q,h)) are well-defined under 
slightly weaker conditions, namely when k is not a positive rational and 



h^ -{ak + b) 



for all integers a and b. 



Proposition 3.4. The Jacobi symmetric functions have the following sym- 
metries: 

Jx{u;k,p,q,h)) = Jx{u;k,p' ,q',h), (41) 

fTf 1 h\\ J\{0;k,p,q,h) 1 - 

uj[Jx{u;k,p,q,h)) = ———— T-Jx'{u;k ,r,s,h), (42) 

Jx'{0;k'^,r,s,h) 

e{Jx{u;k,p,q,h)) = Jx{u;k,p,q,h), (43) 

where X' is conjugate to X and all parameters are the same as in Theorem 



Proof. The first two symmetries follow from the previous theorem and for- 
mulas ([35]) . ([36]) . To prove relation ([13]) it is enough to show that the 
automorphism ()34p has an upper-triangular form in the monomial basis. 

We prove this for the following more general automorphism 6q r defined 
by 

Oq,riPi) = Pi + rq\ (44) 

15 



where r and q are arbitrary parameters. For a given partition A consider the 
set of all different partitions /i, which one can get from A by eliminating at 
most one part of it (or one row in the Young diagram representation). We 
have the following obvious formula: 

mx{xi,...,Xn,q) = ^ q''m^{xi,...,Xn), (45) 

tiU{a)=X 

where (a) denote the row of length a. Iterating this formula N times we get 
mx{xi,X2,...,Xn,q,---,q) = ^^ q^^^'^^^Nx,^m^{xi, ..., Xn), (46) 

where fi runs over all partitions, which can be obtained from A by removing 
some of its rows, and A^a,/^ is the number of sequences of nonnegative integers 
ai, . . . , Cat such that n U (ai) U • • • U (oat) = A. 
Now we note that if r is a positive integer then 

0q,rimx) =mx{x,q,...,q), 

where q occurs r times. The triangularity of Oq^r in that case follows from 
(j46p . Since the coefficients are polynomial in r this is true for all r, which 
completes the proof. D 

To define higher order integrals of BCoo CMS system we need the fol- 
lowing infinite dimensional version of the shifted BCn polynomials [12]. 
Consider the projective limit 

A('=''^) = limAS'") 

taken in the category of filtered algebras with respect to the homomorphisms 
sending the last variable to and the filtration defined by the total degree of 
polynomial. By definition any element / from A^'^''^^ is a sequence of elements 
/at S A]y' such that /jv4.i(2:i, . . . ,Z]\[,0) = /^{zi, . . . ,zn) and the degrees 

of polynomials /at are bounded. We can evaluate element / G A]^' at any 
infinite vector z = {zi,Z2, ■ ■ ■) with finitely many nonzero coordinates. 

Proposition 3.5. (Okounkov, Olshanski [12]J For any partition A there 
exists Ix € A^ ' ' of degree deg{Ix) = 2|A|, which satisfies the conditions 

Ix{^Ji,k,h) = 

for any partition /i such that A ^ /i, and the normalisation condition 

Ix{\k,h) = C^{l)Cl{2h-l). 

Proof. Consider the sequence of polynomials {Ix{zi, . . . ,ZN,k,h)}]yyi(^x)- From 
formula (124^ we have 

Ix{zi,...,ZN,0,k,h) = Ixizi,...,ZN,k,h), 

so this sequence defines an element from Ix{k,h) with required properties. 

D 
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The following theorem gives an infinite-dimensional version of the Harish- 
Chandra honiomorphism. 

Theorem 3.6. For any f € A*-'^''^-* there exists a unique differential operator 
£\ ,p,<i, ^^ ^ such that for any positive integer N the following diagram is 
commutative 

£{k,p,q,h) 

A ^ — > A 

An ^ Ajv 

where h = —kN — ^p — q, ip^ is defined by [27\ ) and the differential operators 
C\ are defined by the Harish- Chandra homomorphism [2^). 

Proof. Define the operator Cr '^''^' by its action on the Jacobi symmetric 
functions: 

jCfP^i^''\j^{u;k,p,q,h)) = f{X)Jx{u;k,p,q,h). 

If /i = —kN — 2P ~ Q then from Theorem 13.11 we see that 

ipN{Jx{u;k,p,q,h)) = Jx{ui, . . . ,UN,k,p,q) 

Therefore the operator CK '^'''' reduces to the operator CK in A^v with 
the property 

-(N) 



L) '{Jx{ui,...,UN;k,p,q)) = f{X)Jx{ui, . . . ,UN;k,p,q). 

(A 
/ 



By Harish-Chandra homomorphism ()22p Cr is a differential operator of 



order equal to deg{f). 

Let us prove now that jCf is also a differential operator of the same order 
I = deg{f). By definition it is enough to prove that 

[[[£f,gi],g2]...,gi+i] =0 

for ah 51 , ... , gi+i G A. Fix 51 , ... , gi+i and denote [[[£/, 51] , 52] ••• , gi+i] as 
D = D[h] G A. Express 

D = '^cx{h)mx 

A 

as a finite linear combination of monomial functions mx- Let M be the 
maximum of the lengths of all the corresponding partitions A. Since for all 
N > M the images (pN{'m\) are linearly independent and ipj\f{D) = for 
h = —kN — \p — q, the corresponding coefficients must be equal to for 
infinitely many values of h. Since they are rational functions of h they must 
be equal identically to zero. D 
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4. Fieri formula and SC-invariant ideals 

In this section we describe all the ideals / C A, which have a linear basis 
consisting of Jacobi symmetric functions. Since this is equivalent to the 
invariance of the ideal / with respect to all quantum integrals £ r ' ' of 
BCoo CMS system we will call such ideals BC-invariant (cf. |17j). 

We start with an infinite dimensional analog of the Fieri formula, which 
in the BCn case was found by van Diejen [5]. We should mention that in 
the form (jlSp Fieri formula contains many zero summands. The following 
lemma shows that the number of the non-zero coefficients in (jlSh for large 
A^ depend only on the diagram A (but not on A^). 

Lemma 4.1. Let L, J, a be the same as in Theorem \2.4\ and M{L) and 
M{J) he the maximal elements of L and J respectively. If X + a{J) is a 
partition and M{L) > /(A) + |L| + 1, M(L) > Af(J) + [L| + l thenV,^.j,{\- 

p) = 0. 

Proof. Let / = M{L) and p be the maximal nonnegative integer such that 
{/, / — 1, . . . ,/— p} C L. By assumption we have l—p— 1 > M{J) + \L\ —p > 
M(J), so I + 1 ^ J and I — p — 1 ^ J. This means that V^,j^ .^(A — p) with 
p given by (fTTj) contains a factor 

/ 
A = v{aixi + xij^i)v{ai^pXi-p - xi-p-i) JJ v{ai^iXi^i + aiXi) 

i=l—p—l 

where Xi = Xi + ki + h, i = 1, . . . ,N and h = —kN— 2P~Q- Let us show that 
for any choice of (T^, r & L the product A is equal to 0. By assumptions we 
have l—p—1 > l{\), therefore x;_p = k{l — p) + h, Xi_p_i = k{l — p — l) + h. 
So if ai^p = 1, then v{ai-pXi-p — xi-p^i) = v{k) = 0. The same arguments 
show that if ai = —1 then v{aixi -|-x;+i) = 0. Therefore we can assume that 
ai-p = — 1, o"i = +1. But in this case there exist i such that I —p+l>i>l 
and (Tj-i = —1, o"j = 1 and f (a"j_iXi_i + aiXi) = 0. D 

Note that the inequalities of Lemma 14.11 are automatically satisfied if 

M{L) >/(A)+2r + l, 

where r = |L| + \J\. 

Now let us fix a natural number r, a partition A and two disjoint finite 
sets L, J C N such that A + a{J) is a partition, M{L) < /(A) + 2r + 1 
and \L\ + \J\ = r. Let us fix also the set of signs a{L),a{J). Note that if 
A^ > 1{X) + 2r + 1 both sets L and J belong to / = {1, . . . , N}. 

Lemma 4.2. There exist unique rational functions ^O^,-) ja(^)' ^a(j)x(^) 
such that for any set I = {1, . . . , N} with N > /(A) + 2r + 1 we have 
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The rational functions V\j-. Ah), VZ^^ j \{^) have the following form 

K{.J),x^h) = Qtih)X{{\j+k3 + h--aj{p-l)){\j+kj + h + aj{-p + q-k)), 
jeJ 

^a(LlJxW = Qx{h) Y[{Xi + kl + h--aiip-l)){Xi+kl + h+ai{-p+q-k)), 

where Q'^{h),Q^{h) are some rational functions of h with poles and zeros 
of the form i(a + kb) with some integers a, b. 

We also need the following 

Proposition 4.3. The value of the Jacobi symmetric functions at zero is 

J^(0) - ,\.\ Cl[h + \p + q)Cl{k + h-\p+\) 

^'^^~ C-{-k)Ct{2h-l) ■ ^^'^ 

The proof follows from the finite-dimensional formula (J17p . 

Now we are ready to give an infinite dimensional version of the Fieri 
formula. Recall that the parameters p, q.k are generic if p, q, k and 1 are 
linearly independent over Q. We say also that h is admissible if 

h^ -{ak + b) 

for all non-positive integers a and b. We know that under these conditions 
Jacobi symmetric functions are well-defined. 

Theorem 4.4. For generic p, q, k and admissible h Jacobi symmetric func- 
tions satisfy the following Fieri identity 

2^ErJ, = E(-l)'^'^.W,7,A(M^.t^),A(^);^;^f^ (48) 

where Ej. is the r-th elementary symmetric function and the sum is taken 
over all a{L),a{J) with L,J C N being the disjoint finite sets such that 
\L\ + \J\ = r, M{L) < /(A) + 2r + 1 and A -|- a'{J) is a partition. 

The proof follows from lemmas 14.11 14.21 and formula (j47p . Note that all 
coefficients have a natural regularisation for all admissible h due to some 
cancellations of the common factors in the numerators and denominators. 

In the case r = 1 we can make the formula more explicit. 

Corollary 4.5. Jacobi symmetric functions satisfy the following Fieri for- 
mula 

2pAn)Mu) = J2v{^,h,P) (^-^^^J^^f,{u) - Mu)^ , (49) 
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where Ei = pi is the first power sum, 
V(\h ^ _ ' n' ^^' ~ ^^ + ^^' ~ ^ ~ ^^^^^' + ^^ + ^^' +3-l)+ 2h) 

(A^ + k{i-l) + h + y + q){Xi + k{i + ^(A) + 1) + 2h){\i + ki + h-y+^) 
(Ai + k{i - /(A) - 2))(A, + ki + h){X, + ki + h+ i) ' 

Vf A h -f) = U (Ai - A,- + k{i - j + 1))(A. + A,- + k{i + J + 1) + 2h) 
^ ' ' '^ y (A,-A,- + A:(i-j))(Ai + A,- + A:(^ + j) + 2/i) 

(A, + k(i + l) + h-lp- q){Xi + /c(2 - l{X))){Xi + ki + h + ^ - D 
{Xi + k{i + 1{X) + 1) + 2h){Xi + ki + h){X, + ki + h-^) ' 

and the sum is taken over all (3 = ibej, i = 1,2,... such that X + (3 is a 
Young diagram. 

We can rewrite it in combinatorial way similar to Macdonald (see [|10j.VI.6]) 
using the following "content" functions (cf. ()14p - (jl6p l: 

cl{U,x) = Xi + j + k{X'j + i)+x, (50) 

cl{U,x) = Xi-j-k{X'^-i)+x, (51) 

cl{U,x)=j-l + k{i-l)+x. (52) 

Let /i = A + ffj be the Young diagram, which is the result of adding to A 
one box in i-th row. Consider the rectangle (/(A) + 1) x (/(A') + 1) and the 
vertical strip vr,, consisting of boxes in this rectangle belonging to the same 
column as added box (excluding the last one). 

Then the coefficients in Fieri formula (1491) can be rewritten as 



y(A, K e,) = Kti^, h) W i?+(n, /i), (53) 



c^(D,l)c+(D, 2/1-1) 

c;:(n,i)c+(n, 2/1-1)' 



where 

Bii^^ h) = :t]~'v:x)~'z :: , (54) 

^ c° (□, h + ^p + g))cO (□, kjljX) + 2) + 2/i)c° (□, /, - ip + ^ + fc) 

^^ ' ' cO(Q,/i + A;))cO(m,-A;(/(A) + l)cO(Q,/i + i + A.) 

(55) 
and □ denotes the added box in the i-th row. 
Similarly, 

V{X,h,-ei) = K-{[I],h) Yl B^{n,h), (56) 

□GTTi 

where 

T-r c-{n,2k)c+{D,2k + 2h) , ^ 

B-(a,h)= TT ^^ ' ^ ^\ ; -^, 57 

' ii^ c,-(n,o)c+(n,2/.) 
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^_ ^ ciJB, 2k + l + h-\p- g))cO (□, 1 + fc(l - KA)))c^(a, fc + /i + Ip + i; 

^^ ' ^ cO(Q,l + A: + /i)cO(Q,l + A;(/(A) + 2) + 2/i)cO(Q,/i + i + A:) 

(58) 
H = \ — Ei and □ denotes the box taken out from the i-th row. 

Comparing with Macdonald's form of Fieri formula for Jack polynomials 
(see formulas (6.24) in [TO]) we see that the vertical strip ttj goes beyond 
the diagram A within some rectangle containing this diagram. The choice 
of this rectangle is actually not very relevant: one can check that for any 
rectangle M x N containing the Young diagram // = A + ej (in particular, 
with M > /(A), N > /(A')) we have the same formulas (I53]),(I5SD with 

^^ ^ c° (□, h + ^p + g))c° (□, k{M + 1) + 2h)cl{B, h - ^ + ^ + k) 

' cO (□,/i + A:))cO(Q,-A;M)cO (□,/! + i + A:) 

(59) 
^_ ^ c°(B, 2k + l + h-^p- g))c° (□, 1 + fc(l - M))cl{B, fc + /i + Ip + 1) 

^^ ' ' cl{B,l + k + h)cl{B,l + k{M + 2) + 2h)cl{B,h + ^ + k) 

(60) 

Now we are ready to describe all i?C-invariant ideals / C A. It turns out 

that in contrast to the ^„-case [17] these ideals admit a simple description. 

Theorem 4.6. Let parameters p, q, k be generic and h be admissible. A 
non-trivial BC -invariant ideal in A exists if and only if 

h = —km — n p — q, h = —k{m + 1) — n p -\ — (61) 

or 

h = -k{m -^2)-{n+l) + -p + q, h = -k{m +l)-n+-p- - (62) 

for some non-negative integers m and n, in which case it is unique. 

The corresponding ideal is linearly generated by the Jacobi symmetric 
functions J\ with A containing the rectangle {m + 1) x (n + 1). 

Proof. Assume first that h is not of the form (|61|) , (|62|) for all non-negative 
integers m, n. One can check that under our assumptions on the parameters 
this implies that 

V{X,h,-e,)^f^^O. (63) 

Let I be any SC-invariant ideal in A, which by definition means that / 
is a linear span of Jacobi symmetric functions J7a 

/ = Span{J\, A G J7}, 

where is some set of partitions. 

If O is empty set then by convention 1 = 0. Suppose that J7 is not empty 
and A G il. If A = then / = A. If A 7^ then since piJx^I according to 
Fieri formula (I49p and ()63p there is a partition // C A such that J7^ G / and 
^ is obtained from A by removing one box. Therefore by induction in this 
case we have that G $7 and / = A. 
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Let us now prove that for h of the form (|61|) . (|62|) the i?C- invariant ideal 
does exist and unique. According to proposition 13.41 (more precisely, the 
dualities ([^T]) and ([^3]) ) it is enough to consider only one of those cases. 

So we can suppose that h = —k{'m + 2) — {n + 1) + ^p + q iov some non- 
negative integers ttt., n and consider the set Q. of all partitions, which contain 
(?7i + 1) X (n + 1) rectangle Rm+i,n+i- Let / = Span{Jx), A € fi. We claim 
that this is an ideal. It is enough to show that this subspace is invariant 
with respect to multiplication by Er for all r G Z_|_. From Fieri formula 
([18|) we see that this will follow if we prove that VZj^^ jx^^^^JiJ) a(^) ~ *-* 
if A contains i?m+i,n+i and A + o"( J) does not. But in that case we have 
Am+i = re + 1 and an+i = —1 and therefore by Lemma [12] ^^Ij) a(^) ~ ^' 
This proves the existence of the ideal. 

Now let us prove the uniqueness. Let J be an invariant ideal such that 
J ^ 0, A and J^\ G J. If A does not contain the rectangle i?m+i,n+i then 
we can apply Fieri formula (j49p and as before we will get 1 G J. Therefore 
A D Rm+i,n+i and J C I. Using again the Fieri formula we can get any 
subdiagram of A containing the rectangle Rm+i,n+i, which implies that J = 
/. ' D 

We will describe explicitly the corresponding homomorphism A — > A/I 
using the following infinite dimensional analogue of Okounkov's binomial 
formula. 

Proposition 4.7. The infinite dimensional Jacobi symmetric functions can 
be written in terms of Jack polynomials as follows 



Mu-Xp,,M =^^^'^'jj^f^fj^/.iu:-k) (64) 

where Pf_i{u; —k) is Jack symmetric function corresponding to the partition 
fi, I^ is the interpolation function defined in Proposition |g.5l 

The proof follows directly from Theorem 12.61 by the same arguments as 
in the proof of Theorem 13. 6[ 

Remark. Using the difference version of Okounkov's formula Rains [15] 
introduced the Koornwinder polynomials of infinitely many variables. The 
formula (j64p shows that Jacobi symmetric functions are the limiting case of 
these polynomials. 

Proposition 4.8. When 

[h + km + n -\ — p + q){h + k{m + 1) + n H — p ) = 

the corresponding ideal is the linear span of Jack polynomials P\, where A 
contains the rectangle of the size [m + 1) x (n + 1). The corresponding 
homomorphism is given by the formula 

m n 

^mAPa) = ^ut + k~'J2'^j (65) 

i=l i=l 
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In the case when 

{h + k{m + 2) + {n + l)--p-q){h + k{m + I) + n - -p + -) = 

the corresponding ideal is the linear span of the polynomials 6{Px), where 
X contains the rectangle of the size [m + 1) x (n + 1). The corresponding 
homomorphism is given by 

^mAPa) =f2< + ^"' E< + (-2)"^^^^^^- (66) 

Proof. Because of the proposition 13.41 it is enough to investigate only one of 
the cases. Assume that h = —km — n — ■^p — q for some non-negative m, n 
and consider the set of ah partitions, which contain (m + 1) x (n + l) 
rectangle Rm+i,n+i- It was proven in jTT] that / = Span{P\,\ G 0} is the 
kernel of ip defined by (j65p and therefore is an ideal. We claim that it is also 
i3C-invariant. 

More precisely, we claim that / = Span{J\,X G J]}. To prove this it is 
enough to show that for such h in the sum at the right hand side of ()64p 
there are no terms P^{u; —k) with fim+i < n. But this follows from the fact 
that for h = —km — n — ^P — Q 

Cljh+lp + q) _ ClJ-km-n) _ 
C',ih + lp+q) - CO{-km-n) "^ 

if Xm+i > n and fj-m+i <ri. D 

5. Deformed CMS operator of type BC{m, n) as a restriction 

The main goal of this section is to show that the gauged version of 
the deformed CMS operator ^ related to generalised root system of type 
BC{m,n) is a restriction of BCoo operator C^^'P'I'^). This implies quantum 
integrability of this system and allows us to generalise the main results about 
the Jacobi polynomials to the deformed case. 

Recall that generalised root system R of type BC{m, n) is the union 

R = Rreal U Riso (67) 

of real (non-isotropic) roots 

Rreal = {±^1, ±2ej, ±ei±ej, ±6p, ±2Sp, ±6p±6g, I < i 7^ j < m, I < p ^^ q < n} 

and isotropic roots 

Riso = {±£1 ± Sp}, 
where the bilinear form is defined by 

(£»,£,) = 1, {Sp,6p) = -1, {ei,5p) = 0. 

The Weyl group Wq = {Sm x Z™) x {Sn x Zg) acts on the weights by sep- 
arately permuting Si, j = 1, . . . ,m and 5p, p = 1, . . . ,n and changing their 
signs. 
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Consider the deformations of this root system in the sense of [TB]: the 
roots remain the same, but acquire the multiphcities m{a), what changed is 
the bihnear form. In the BC{m, n) case the admissible deformations have 
the form [I6j : 

(ei,ej) = 1, {5p,5p) = k, iei,6p) = 0. 

m(±5p ± Ei) = 1, m{±ei) = p, m{±2ei) = q, m{±5p) = r, 

m{±26p) = s, m(±ej it Ej) = k, m{±6p ± 6q) = k~^ (68) 

with the following relations to be fulfilled 

p = kr, 2q + 1 = k{2s + l). 

The corresponding deformed version of the CMS operator Q has the 
form 

i=l a=l i<j ^■^'' -^^ -^'-^^ ^ ^ 

- ^ ( ' _ ° {xidi - kyada) + ' " _ {xidi + ky^da) ) , 

\Xi IJa Xiyct J- / 

Introduce the new coordinates 

Ui = -{Xi + X-^ - 2), Va = -{Va + Va^ - 2). 

In these coordinates the operator /^i"^'") has the following expression (cf. 

m)- 

m n m 



i=l a=l i<j 

n 



Ui — U 



i f^j 



Va+VfB , -s-^Ui+Vc 



V" — {Vada - Vpdp) - V" — -{Uidi - kVada 



m n 



-{n+kim-l)+p+2q) ^Ui^i + ^ t;„a„ -(l+2p+2(7) [J2di + Y,da 

\i=l a=l / \i=l r=\ 

+2 ^ d,{n,d,) + 2kJ2 daivada) - 4fc ^ ""'"i ~_ ""'"^ 

1=1 a=l i<.j -' 
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Y^ Vgda - V/sdiS y^ Ujdj - kVgda 

^ Va-Vp ^ Ui-Va 

Let Pm,n = C[ui, . . . , Um,vi, . . . , f„] be the polynomial algebra in n + 
m independent variables. Denote by ^m,n,k its subalgebra, consisting of 
polynomials, which are symmetric in ui, . . . , Um and vi, . . . , u^ separately 
and satisfy the conditions 

df , df 

Ui- kVa-^ = 

OUi OUa 

on the hyperplanes Ui = Va for alH = 1, . . . , m and a = 1, . . . , n. For generic 
values of parameter k (more precisely, if k is not a positive rational number) 
this algebra is generated by the deformed power sums 

m n 

Pa{u,v,k) = ^u'i + k-^^v'i, a = 1,2,... 

(see Theorem 2 in p6]). Consider a homomorphism ipm,n '■ ^ — ^ ^m,n,k 
defined by 

Vm,n{Pa) =Pa{u,V,k). 

The main result of this section is the following theorem (cf. Theorem 1 
in[I7!). 

Theorem 5.1. The following diagram, is com,mutative for h = —km — n — 
\p — q and generic values of parameter k: 

r(k,p.q,h) 

A — > A 

i <fm,n i fm,n (70) 

In other words, the deformed CMS operator ^W^ is a restriction of the BCoo 
CMS operator onto the corresponding subvariety T>{m,n,k) = SpecAm,n,k- 



Proof. The proof is similar to the proof of Theorem (j3.ip . Let us split the 
operator C^"^'^' as the sum: 

where 

m n m 

Ui + Uj 



^(m,n) ^ ^(^^a^)2 + k Y,{Vo.d^? " ^ E ^^^(^^^^ - U,d,) 

a^ a j 
1=1 a=i i<j ■' 

_ y-v _a ^{y^Q^ _ vadp) - V] — -{uidi - kvada) 

^„Va-Vi3 '^ ^-^ Ui-Va 

/ m n \ 

-{n + k{m- l)+p + 2q) "^Uidi + ^Vada , 
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1=1 a=l i<C] ■' 



^„ Va-Vf3 '-^ Ui-Va 

a<f3 ^ t,a 

(■m n \ 

i=l r=l / 

The claim follows from the following formulas: 

a-l 

A'^"'''^\pa) = (1 + k)a'^Pa - ka^PrPa-r - a(n + k{m - 1) + p + 2q)pa, 

r=0 

A^"''''\paPb) = ^('"•")(Pa)p, +p,^(™'")(p,) + 2abpa+b, 

a-l 

^Kn)(^j = 2(1 + A:)a2p„_i - 2A:a J^p,pa-,_i - a(l + 2p + 2q)pa-i, 

r=0 
B^'^-'^HpaPb) = B^^^^\pa)pi>+PaB^'^^'''>{Ph) + ^abpa+b-L 

where pa = Pa{u,v, k) and po = m + k~^n. They can be checked directly 
with the use the following additional identities: 

111"' 77 

^{Ui +Va)— = ^ {pr{u)pa-r{v) + Pr{v)pa-r{u)) , 

i,a * " r=0 

U? — U" 



Y] = Vpr(n)pa-r-l(w) 

^-—^ 11,: — 71. ^— ' 



. Ui-Va „ 

where po(^) = "^) Po(^) = '^• 

Note that from these formulas it follows that for generic k the operator 
£m,n j-Q^pg f^Y^Q algebra Am,n,k into itself. The rest of the proof is the same 
as in Theorem 13.11 D 

Remark 1. One can show that the diagram in the theorem is commu- 
tative and the operator C^'"- preserves the algebra A^ „ ^ for all values of 
parameter k (see [17 1, where this is proved in the A{m,n) case). 

Remark 2. Because of the symmetry ()35p the theorem remains true 
when h = —k[m + 1) — n — gP + 2 if we change the parameters in the 
operator Z^"*-" according to the formula (|38|) . The situation for other two 
exceptional values ()62p of parameter h from the theorem 14.61 still needs to be 
understood. We believe that the restriction is again the deformed BC{m, n) 
operator written in a different coordinate system. 

As a corollary we have quantum integrability of the deformed BC{m, n) 
system. Let the algebra A^ ' ' and the operators C'^^'P'I'"-' be the same as in 
Theorem 13.61 from section 3. 
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Corollary 5.2. For h = —km — n — \p — q and any f G A'-'^''*) there exists a 
unique differential operator Cy"'^' on Am,n,k such that the following diagram 



Ak,p,q,h) 



is commutative 

A ^^' — > A 

i fm,n i ^m,n 

The operators £y^'"-' commute with each other. 

In the next section we discuss the corresponding algebra of quantum in- 
tegrals in more detail. 

6. Algebra of quantum integrals of the deformed BC{m, n) 

SYSTEM 

Consider the deformed BC{m, n) root system (|68p and choose the follow- 
ing system of positive roots: 

^teai = i^i^ 2ei, Ei ± ej,6p, 25p, 5p±5q, l<i < j <m, l<p < q<n}, 

RL = {^^^^}■ (71) 

Let p be the corresponding deformed version of the half-sum of positive 
roots: 



P= 2 Yl "^(«)«' 



which in our case has the form 

m 



p = '^l-p + q + k{m -i)\ei + '^{-r + s + m + k ^{n-j)j6j 

m n ^ ^ 

= -Y^{h + n + ki)e,-k-'J2{h + -k--+j)6„ (72) 

i=i j=i 

where h = —km — n — -^p — q. 

Consider now the following deformed analogue of algebra Ajy' from sec- 
tion 3. Let Am,n C Pm,n = Cf^i, . . . , Wm, zi, . . . ,Zn] be the algebra of poly- 
nomials in wi, . . . , Wm ,zi, . . . ,Zn, which are symmetric separately in vari- 
ables (wi + h + n + ki)"^, i = 1, . . . ,m and {zj + hk~^ + ^ — ^k'^+jk~^)'^, j = 
1, . . . ,n and have the following property: 

f{w -ei,z + 6j) = f{w,z) 

on each hyperplane Wi + k{i — 1) = kzj -|- j — n for alH = 1, . . . , m and j = 
1, . . . , n. In terms of the deformed root system BC{m, n) the last condition 
can be rewritten as f{X + a) = f{X), when X = (w,z) belongs to the 
hyperplane 

{X — p,a) -\ — (a, a) = 
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where a = 6j — £i is an isotropic root. The algebra Ivin]n is isomorphic 
(by the shift by p) to the algebra I^r^b introduced in [16] . when R is the 
generalised root system of type BC{m,n). 

Let us denote by Hm,n the set of partitions A such that Am+i < n. 
This means that the corresponding Young diagrams are contained in the 
fat (?n,, n)-hook (see |16j). For such a diagram one can consider two sub- 
diagrams fi and v, where i^ consists of first n columns of A and fi is the 
remaining part. Consider the following embedding x '■ Hm,n -^ C"^™", using 
the formulas 

x{X) = {w{X),z{X)), (73) 

where 

WiW = fii, i = l,...,m, Zj{X) = ly'j, j = 1, . . . ,n. (74) 

In other words Wi is the length of i-th row of fi and Zj is the length of j-th 
column of A G Hm,n ■ 

It is easy to see that the image of x is dense with respect to Zarisski 
topology. Therefore considering any element from A' ' -* as the function on 
Hm,n we have well defined map 

Theorem 6.1. For generic values of parameter k the image of the restric- 
tion homomorphism reSm,n coincides with the algebra Am,n ■ 

Proof. Let us consider the following functions 

Mw) = J2^B2i{wi + h + ki + 1/2) - B2i{h + ki + 1/2)) 
i>l 

where B2i{x) are the even Bernoulli polynomials. These polynomials have 
the following properties (see e.g. J19j): 

B2i{x) = B2i{l - x) (75) 

and 

t 

B2i{x + t) = 2l ^{x + i - if-^ + B2i{x) (76) 

j=i 
for any positive integer t. 

It is easy to check that the functions fi{w) belong to A^^''^' and generate 
this algebra (cf. [TT], section 3). When w = A is a partition we have 



fi{X) = 2/ ^ (j - 1 + A:(i - 1) + /i + A; + i)2 



\2l-l 

2' 

fe)6A 



On the other hand, consider the functions 



J I {w,z) 



"" / 1 1 \ 

^[B2i{wi + h + n + ki + -)-B2i{h + n + ki + -)\ 



n 
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n 

+k^^-^^{B2iizj + k-^h + k-^j - -k-^ + 1)- B2i{k-^h + k-^j --k~^ + 1)). 

J=l 

It is easy to verify that these functions belong to Am',n and generate it for 
generic values of k. If Wi = fii, Zj = u', are determined as above by some 
partition A we have 

/r'"(A) = 2/ Yl {j + n-l + k{z-l) + h + k + ^f~' 

+2/A;2'-i Y^ (i - 1 + k-\j - 1) + k-^h + -k-^ + l)2'-i, 
which can be rewritten as 

(i,i)GA 

and coincides with /((A). Thus we have shown that 

' (ii'm,n\Jl) — J I ! 

which implies the theorem. D 

As a corollary we have the following deformed version of the (inverse) 
Harish-Chandra homomorphism. Let 2?m,n be the algebra of quantum inte- 
grals of the deformed BC{m,n) system with the Hamiltonian C™-'^, 

Corollary 6.2. For h = —km — n — ^p — q and generic values of k there 
exists a monomorphism 



\2l-l 
2' 



An alternative version of this monomorphism was established in [16] by 
a complicated recurrent construction of the quantum integrals. The present 
form gives a more conceptual way to do this. 

Now we are going to describe an important linear basis in algebra Km]n , 
consisting of the deformed interpolation EC polynomials I\{z, w, k, h). They 
can be defined as follows (cf. Proposition 12. 5|) . 

Proposition 6.3. For any partition A such that Xm+i ^ n there exists a 
polynomial I\{zi, ...,Zn, wi,..., Wm, k, h) G A^i,™^ of degree deg{I\) = 2|A|, 
which satisfies the following conditions: 

Ix{z{fj,),w{fj,),k,h) = 0, 

where fj, G Hm,n is any partition such that A ^ ^, z{^) and w{fi) are defined 
by |73| j, and normalisation 

h{z{X),w{X),k,h)= l[ {i + Xi-j-k{X'j-i)){2h-l + X,+j + k{X'j+i)). 
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The proof easily follows from Proposition [ 

One can give the following deformed analogue of Okounkov's combinato- 
rial formula for interpolation BC polynomials [11]. 

Let T be a reverse bitableau of type (n, m) and shape A. We can view T as 
a filling of a Young diagram A by symbols l<2<---<n<l'<2'<---< 
m' with entries decreasing weakly downwards in each column and rightwards 
in each row; additionally entries 1,2... ,n decrease strictly downwards in 
each column and entries l',2'...,m' decrease strictly rightwards in each 
row. Let Ti be a subtableau in T containing all symbols 1', 2' . . . , m' and 
Tq = T — Ti and /i is the shape of Ti. 

Introduce the polynomials 

ix{zi,...,Zn, wi,...,Wm,k,h) = "^ipxi-k) Jl /r(n), (77) 

T DeA 

where 

Ma) = (zT^o) +h + kT{U)f - {clip, 0) + /i + kT{U)f 

if T(n) = 1, . . . n and 

friU) = k\w\T(n)\+h+n+k~^\T{U)\f-{k-^cl{U,^)+h+n+k-^\T{U)\f 

if T(n) = 1', . . . m' in which case we denote |T(n)| = 1, . . . m respectively. 
Let ipTi—k) be the same weight as for super Jack polynomials (see formula 
(41) in [T7]). 

Proposition 6.4. The deformed interpolation BC polynomials can be given 
by the following formula: 

Ixizi,...,Zn,Wi,...,Wm,k,h) = dxix'{zi, . . . , Zn,Wi, . . . ,Wm, k'^ ,h) 

where Ix{z, w^k^h) is given above by combinatorial formula ([y7| j and 

d, = (-l)l^lA;2W-^^"(^^ 



Cxi-k) 

The proof is based on the Okounkov's results [llj and is similar to the 
super-Jack case (see Theorem 12 in [17J). The appearance of the dual pa- 
rameters and transposed Young diagram is a consequence of our choice of 
coordinates. 

7. Super Jacobi polynomials 

Li this section we introduce and investigate the deformed version of the 
Jacobi polynomials. In particular, we prove the corresponding analogues of 
Pieri, Opdam's and Okounkov's formulae. 

We define the super Jacobi polynomials Jx{u-, v; k, p, q) as the image of the 
Jacobi symmetric functions J^xiz',k,p,q,h) under the homomorphism (j65p : 

Jx{u,v;k,p,q) = ipm,niJxiz;k,p,q,h)), (78) 

where h = —km — n — ^p — q. 
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From Theorem 15.11 and Proposition 14.81 it follows that J'x{u,v;k,p,q) are 
the eigenfunctions of the deformed operator £™'" and form a basis of Am.n 
if we consider all the partitions A G Hm,n from the fat (m, n)-hook (i.e. with 
Am+i < n). 

It is convenient also to use the normalised version of super Jacobi polyno- 
mials. To define them we introduce the following partial order on the pairs of 
partitions: (/i, i^) :< {jl, v) if the corresponding X = (xi, . . . , Xm+n) = {^^ /^) 

with Xi = iyi,...,Xn = Vn,Xn+l = fJ-l, . . . , Xm+n = fJ-m and X = (P, /x) 

satisfy the inequalities 

xi-\ \-Xi < xi-\ \- Xi (79) 

for all i = 1, . . . ,m + n. Note that in these formulas v goes first. One can 
check that the highest term in the super Jacobi polynomial J'\{u,v; k,p,q) 
with respect to this order is mfj_{u)mu{v) with the coefficient given explicitly 
by the following 

Proposition 7.1. The highest term in super Jacobi polynomial Jx{u, v; k,p, q) 
has the form a\m^{u)my{v) with 

a, = (_l)M2lAlSil^^^, (80) 

where as before the sub-diagram v C X consists of the first n columns and 
fj, = X\u. 

The proof follows from a similar result for super Jack polynomials. Recall 
that super- Jack polynomial SPx{u, v; —k) is defined as the image of the Jack 
symmetric function P\{x; —k) under the homomorphism '^rn,n (see [17]). 

Lemma 7.2. The highest term of the super Jack polynomial SPx{u,v; —k) 
has the form h\m^{u)mi,{v) with b\ = a;^2~l ' and a\ given above by ( fgOj) . 

This follows directly from the definition of the function C^ : 
Ca"(^)= n (A^-i-A:(A;■-^)+x), 

the formulas (7.11'), (7.13'), (10.10), (10.11) from Macdonald [lOj and for- 
mula (25) from |17j . 

Now we define the normalised super Jack polynomials as 

SPl{u,v--k) =bl^SPx{u,v;-k). 

and the normalised super Jacobi polynomials as 

Jxiu,v;k,p,q) = a^^Jx{u,v;k,p,q). 

We have the following generalisation of Okounkov's formula to the de- 
formed case. 
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Theorem 7.3. The normalised super Jacobi polynomials can be written in 
terms of super Jack polynomials as follows: 

TAu vkva)-^ JmhMX)AM2RsP*iu V -k) (81) 

where A € Hm,n, h = —km — n — ^p — q and ^7^(0) = J^{0, 0; k,p, q). 

The proof fohows from the fact that the super Jack and corresponding 
super Jacobi polynomials have the same highest term with respect to the 
partial order introduced above. 

We are going to present now an explicit expression for the values J7'^(0). 
Let 

X(A) = zi{X)6i + ■■■ + Zn{X)Sn + wi{\)ei + ■■■ + w,n(A)e„, (82) 

where Zi{X), Wj{X) are defined above by ([75]) . and p be the deformed version 
of half-sum of positive roots given by (I72p . Introduce 

-v-M ,\ r((x(A) -/),aV) -m„-im„/2)r((-p,a^) -im„/2) 

r((x(A) - P, a^) - 2^a/2W{{-P, a^j -ma- 2^a/2) 

where the deformed inner product (,) and the multiplicities are given by 
(pD and a^ = -^ (cf. formula (^ above). 

We have the following generalisation of Opdam's formula (fT3|) to the super 
case. 

Theorem 7.4. The value of the normalised super Jacobi polynomials at 
is given by 

j;:{0,0;k,p,q) = J] T(A,a,A;), 

a£R+ 

where the product is taken over the positive roots (71). 

Proof. We split the positive roots into three groups: 

^t = {^P^ 25p, 5p±5q, l<p <q<n}, 
Rf = {ei, 2ei, ei±ej, 1 <i < j < m}, 

SO that 

Yl T{X,a,k)= Yl T(A,a,/t) JJ T(A,q,A;) JJ T(A,q,A;). 

°6-R+ a&RJ aeR+ "£/?+„ 

Using Proposition 12.31 we have 

C?,{-km)C^Jx + n) 



n T(A,a,A.)=4l'^l- 



C;i{-k)C^{y + n-iy 

a&RJ 

n T(A ak\ = ( 1)1-1 g°(-n)C°(x)CO(|/ + fc) 

i~\ ^ ' ' ' ^ > C-{l)C+{y-n-l)C^,{y + k + kmy 
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where 



X = k{l — m) — n — p — q-\ — , y = —n — 2km — p — 2q. 



We have also by definition 

n riX,a,k) = f\f\-^^^-!^^^^ , , J + k^ + y-l 

^\ M -Li ki^i - fii+j-ki-n-lkv'. + ^i+j + ki + y-l 

From proposition 14.31 we have 

IM 1 C^(-km - n)C^(x) 



C-i-k)C+iy-n-l) 
The theorem follows from the following two identities: 

axC-{-k) C^ (-A;)C.- (1) fj f}^ ku'- - ^i,+ j - ki - n- I 

and 

Cl{x) _ Cl{x + n) CUx)C'Ay + k) 



C+(y - n - 1) C^iy + n - 1) C^(y - n - l)CO(y + A: + A;m) 

m n • I ; • I i 

TTTT j + fc^ + ?/-i 

^\^\ kv'. + ii,+j + ki + y-l' 
Let us prove the first identity. It can be rewritten as 

Cf,{-km)C2{-n)C^{l) \\f\ kiy'.-i2i + j-ki-n-l- 
To prove this we note that 

z + k{i — m — 1) _ T-r z — kj 



n^ -t l^{l — III — i^) _ -TT 
z + k(i-l) "-I--I- 



.^^ z + A;(i-l) ±lz + /c(/-j) 

for any two nonnegative integers l,m. Using this one can show that 

C^{—km — n) TTTT j — ki — n — 1 

Cf,{-km)C^{-n) " ^ ^ ku'- + j - ki - n - l 

We can also check directly that 

C^I{l)Cu{l) ~ fj U ku'^+ j-ki-n-l ' 

which implies the first identity. 

Now let us prove the second identity. It is easy to verify that it is equiv- 
alent to the following equality 

C+{y + n- l)C+{y -n- l)C',{y + k + km) _^^ j + ki + y - 1 



C+{y-n- l)C70(y + k) f^\ fj^ ku'- + fn + j + ki + y - l' 



4 = 1 J = 
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Using the identity 
I 



n z + k{i- 1) ^ -|-|- z + k{j - 1) 



^ z + k{i-l + m) J-J- z + k{l + i - 1) 

for arbitrary nonnegative integers l,m one can show that 
CO(y + A;(l + m)) _ -|^ " J- + A;i + y - 1 



c^(y + ^) {}■ i~_\ ki^'. + j + ki + y - 1 

A direct check gives also 



C+{y + n-l)C+iy-n-l) -^ " ku'^ + j + ki + y - 1 



C+{y - n - 1) ^^-^ Ai fci/j + j + ^i + fci + y - 1 ■ 

Theorem is proved. D 

Now we are going to present a deformed version of the Fieri formula. 
When n = 1 a related result was found by M. Feigin [8j, who proved the 
bispectrality of the deformed CMS problem ([2]) in that case. 

Consider the set 

O = {±5i, ±ej\ i = l,...,n, j = l,...,m}. 

For a given partition A we say that P G O is admissible if xW + /3 = x(^) 
for some partition A € Hmn- We denote the corresponding partition A as 
A + /?. 

Let R be the BC{m,n) root system (j67p and <, > denote the standard 
inner product: < ej, £j >=< Si, 6j >= when i ^ j, < ei,£i >=< 5j, 6j >= 
1 and < £i,5j >= 0. Introduce the function 

-<^-« -<"•")- n ^^^. <-) 

Q!e-R,<Q!,/3>>0 V ' ' ^ 

where T(A,a,A:) is defined by formula (i83]) . It can be simplified as follows: 

y(. .. ^ ((X(A) - p, n -mp- 2m2MxW - P, ^ -m^ + l) 
^ '^' (/3,/3)(x(A)-p,/3V)((;^(A)-p,/3V) + l) 

TT (x(A) -p,«^)-ma 

where the last product is taken over roots a, which are not proportional to 



Theorem 7.5. The super Jacobi polynomials fTS^ satisfy the following Fieri 
identity 

where the sum is taken over all admissible /3 G C 
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Proof. Let /3 = zbcj, then a direct check shows that (|85p can be rewritten as 

ViX,±ei) = K^{B,h + n) H B^{a,h + n) J] ^^P'^)' 

□67r^(m) ne<(m)\7r;(m) 

(87) 
where 

fi is the same as in ([7i|) . 5^ (D, /i) and K^ (□, /i) are given by (jM|) . ([57|) and 
(j55|) . ([58]) . □ denotes the added box in the i-th row of ^, vr^([II) is a vertical 
strip consisting of the boxes in the minimal rectangle containing Young 
diagram fi+Si, belonging to the same column as added box but excluding the 
last one, 7r^(E]), vr' (□) are similar horizontal strips, 13^ {0,h) = By{\I\,h) 
is given by ([Ml) .([57 |) . ([50]) . ([5T]) . ([52]) with A replaced by its conjugate, k by 
k~^ and h given by ([39l) . 

Similarly for f3 = ±5j we have 

vix,±6,) = ktiB,h+ki{fi)) H Bt{a,h+ki{fi)) H B^{a,h), 

ne<(m) ne7r;,(S)W(S) 



where r is the Young diagram A without the first /(/i) rows, □ is the added 
box in the j-th column of r, i^^(Q,/i)) = K^,{[I],h) is given by (f55]) . (f58]) 
with T replaced by its conjugate, k by k~^ and all other parameters given 

by dMD. 

From the infinite-dimensional Fieri formula (I49D with 



V{X,h,±£i) = K^{B,h) Yl B^{a,h) (89) 

it follows that it is enough to prove the following 

Lemma 7.6. Let A, fi and r be as above. We have the following identity 
valid for any h 

K^{B,h) n B^{a,h) = K^{B,h+l{T')) n B^nh), 

(90) 
when the box □ is added to fj,. // □ is added to r then we have 

KtiB,h+klifi)) H Bfin,h)=K^iB,h) H Bfin,h). 

ne7rA(E)W(m) ne<(m)\<(H) 

(91) 



Proof of lemma. From Fieri formula ()49p and the symmetry of Jacobi sym- 
metric functions (1421) we have 



K^{B,h) n B^in,h)=k-'k^iB,h) n ^^(°'M 

□e7r;^(E]) nG7r^(m) 
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and similarly 

k±(q,/i+/(t')) n B^{u,h+i{T'))=k-^m[ii,h+i{T')) w B^{u,h+i{T')). 

Dividing these two equalities and taking into account that 

B^{U,h)=B^{U,h + l{T')) 

if D G 7r^([Il), and 

B^{U,h) = B^{U,h + l{T')) 

if D G vr' (□) we get (j90p . The second identity can be proved similarly. This 
proves the lemma and the theorem. D 

One can interpret the formula as the bispectrality between the deformed 
BC{m, n) CMS operator ([2]) and the following deformed rational Koorn- 
winder operator 

v(-^'^'> = ^Wiz,P)iTp-l), (92) 

where 

m 
Z = 

i=l i=l 

Tp : f{z) ^ f{z + (3) is the shift operator and 

Tx./ r,-. ^ ((^; /?^) -m0- 2m2/3)((z, /3^) - m^ + 1) -|-|- {z,a^) - mg 

^^' ^^ " (/?, /3)(^, n{{^, /5^) + 1) <„-^^i>o (^' """^ + (^' """^ - ^ ' 

(93) 
where again the product is taken over roots a, which are not proportional 
to (3. More explicitly, 

m n 

p(™.«) = J^[w,+jr,.-i)+w,;,(7;;,i-i)]+^[M^+ (r,,,-i)+w^- (r,r^i-i)], 

(94) 
where Ti^x,Tj^y are the shifts by 1 in i-th x-direction and j'-th y-direction 
respectively, 

w,+ = (1 - ^i^)(i - -^) n(i - ^— )(i - ^— ) 



^Xiei + ^yj5j, 



j^j -■. -J -- ■ -3 



n^ 



7=1 \ ^i + "^yj H 2~ / V "^^ ^i "' 2~ J 

w.:. = (1 + ^)(i + ^) fid + ^)(i + ^) 

Il(i + r^ — iZfc ) (i + r^ — —]^ 
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^'y k^ 2yj '^ 



n^ 



1 



j=i 



yj + k ^Xi + 



1 7- _|_ 2 s 



n 1+ 



1 




As we have mentioned before in the special cases n = and n = 1 the 
bispectraUty was estabhshed earUer in [2] and [8j . 

We would like to emphasize that all the formulas for super Jacobi poly- 
nomials can essentially be found from the usual case simply replacing the 
BC]S[ root system by its BC{m, n) deformed version. This fact is another 
evidence of the importance of the notion of the deformed root systems. 
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